Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix: Issue when the database objects passed to the cleanup task via params aren't JSON serializable #119

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Fix: Issue when the database objects passed to the cleanup task via params aren't JSON serializable #119

wants to merge 1 commit into from
+148 −145

Conversation

ealebed
Copy link

@ealebed ealebed commented Nov 9, 2021

Fix [teamclairvoyant#117]: Issue when the database objects passed to …
8f60d7e 
…the cleanup task via params aren't JSON serializable
@lonyelon
Copy link

Great pull request! 100% solves the problem!

@Limess
Copy link

Limess commented Nov 23, 2021

It'd be great to see this in merged if possible - this is holding us up on upgrading airflow (we could point to this PR but would prefer not to)

@hanizaidi110
Copy link

Hey guys,

I'm getting this error:

Task failed with an exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task self._execute_task_with_callbacks(context) File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks result = self._execute_task(context, self.task) File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task result = execute_callable(context=context) File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 151, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 162, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/vimcar-bi/airflow_dags/admin_airflow_db_cleanup.py", line 275, in cleanup_function airflow_db_model = DATABASE_OBJECTS_DICTS[object_name].get("airflow_db_model") KeyError: 'TaskSet' Marking task as FAILED. dag_id=admin_airflow_db_cleanup, task_id=cleanup_TaskSet, execution_date=20211201T172422, start_date=20211201T172533, end_date=20211201T172533

pceric
pceric reviewed Dec 16, 2021
View reviewed changes
db-cleanup/airflow-db-cleanup.py
@@ -366,10 +370,9 @@ def cleanup_function(**context):
for db_object in DATABASE_OBJECTS:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work if you aren't using the CeleryExecutor as Task and TaskSet don't exist. There needs to be a conditional check like on line 247.

sergey-shambir pushed a commit to ispringtech/airflow-maintenance-dags that referenced this pull request Jan 31, 2022
Applied fix from teamclairvoyant#119
411d5f3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pceric pceric pceric left review comments

@babinin babinin babinin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@ealebed @lonyelon @Limess @hanizaidi110 @pceric @babinin